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Abstract 

As educators increasingly integrate web-based resources into their curriculum, there is a growing 
need for high quality, educationally relevant materials. This study evaluated the Bayou Bend 
website, the result of a collaboration between staff at the Museum of Fine Arts, Houston, and 
faculty and graduate students at the University of Houston to design and develop an educational 
website. Expert designers and usability experts emphasize the importance of the users in 
determining if a website meets the needs and expectations of the intended audience. A sample of 
266 potential users reviewed the Bayou Bend website and submitted an online survey designed 
to measure user perceptions of the design, content, and educational value of the site based on 
criteria identified in a review of the literature. The results of the survey revealed some of the 
strengths and weaknesses of the Bayou Bend site (http://www.bayoubend.uh.edu) as well as 
provided insight into the process of evaluating educational websites. 
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Evaluation of an Educational Website for the 
Bayou Bend Collection and Gardens, Museum of Fine Arts, Houston 
Evaluation, an important phase in the design and development process, is particularly 
critical in the development of websites designed for instructional purposes. The purpose of this 
evaluation was to identify the strengths and weaknesses of the Bayou Bend website in order to 
make improvements and to guide future development efforts by using the design and evaluation 
criteria identified in a review of the literature. While developers may make every attempt to 
adhere to specific criteria during development, ultimately the users will determine how 
successful developers were in their efforts to satisfy these criteria. Accordingly, this study 
addressed the following research questions: 

1 . What are users' perceptions of the design, content, and educational value of the Bayou 
Bend website? 

2. Are there statistically significant differences in user perceptions by user group, user 
experience, user comfort level, or likelihood of a revisit? 

The Bayou Bend Initiative 

In the fall of 1997, faculty from the University of Houston (UH) met with representatives 
of the Museum of Fine Arts, Houston, (MFAH) to discuss how the museum could use a small 
grant to develop technology-based materials that would enhance public access to the arts. Since 
Instructional Technology faculty members at the university were searching for authentic and 
meaningful projects that graduate students could work on as part of their course of study, it was 
decided that the museum would provide the content that students would use as the basis of a web 
design course. In the course, CUIN 7330 - Project-Based Web Design and Development, 
students work in small collaborative teams to design and develop online educational resources 
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using raw materials provided by the museum. Students in the course work with museum 
education staff members to design innovative approaches to presenting the museum's educational 
content over the World Wide Web. 

The first museum/university collaborative project in which UH students used content 
from the museum was the development of a website for Bayou Bend, the American decorative 
arts wing of the MFAH. (The URL for the Bayou Bend website is: 

http://www.bayoubend.uh.edu.) Bayou Bend is a house museum in Houston that contains one of 
the nation's finest collections of colonial era American decorative arts. Bayou Bend, like many 
museums and galleries, was interested in having a web presence. In addition to informing visitors 
of hours and location, a website affords museums an opportunity to share materials with those 
unable to come to the museum (Fulford, 1996; Taylor & Ryan, 1995; Wong, 2000; Zorich, 

1997). Museums are using the Web not only to make their presence known but also to extend 
their educational missions beyond their physical walls (Blumenstyk, 1998; Bowen, 2000; Devine 
& Welland, 2000; Fawkner, 1997; Londono, 2000; Solomon, 1999). 

During the Bayou Bend project, students worked with museum content experts to 
develop site navigation, page design, and educational resources to showcase Bayou Bend and its 
collection. In order to create the website, participants in the course were given access to 
brochures on Bayou Bend's collection, founder, and gardens, and the recently published 
catalogue American Decorative Arts and Paintings in the Bayou Bend Collection. These 
resources provided important information and images that student developers used as the 
foundation of the site. In addition, student teams created searchable databases, produced virtual 
room tours, and integrated streaming media content into the site. A paper entitled A Museum- 
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University Partnership to Develop Web-Based Educational Resources (Robin, Jenkins, Howze, 
& O'Connor, 2001) describes the Bayou Bend web development project in more detail. 

Website Design, Development, and Evaluation 

Although the creation of websites is somewhat limited by the web publishing medium 
itself, the literature suggests that established principles of information and graphic design can 
help to guide the design and development process. Widely accepted publication standards and 
style guides are available to guide the development of content (Lynch & Horton, 1999). 
Numerous books and articles offer suggestions on how to create a website, how to create web 
graphics, and how best to present web-based information (e.g., Andres, 1999; DiNucci, Giudice, 
& Stiles, 1997; Duff & Mohler, 1996; Holzschlag, 1998). In addition, many books and articles 
suggest criteria for evaluating the quality of websites (e.g., Alexander & Tate, 1999; Cooke, 
1999; Pratt, Flannery, & Perkins, 1996). Much of the literature emphasizes the importance of the 
users in the design as well as the evaluation phase. Experts recommend that the design process 
begin by addressing the purpose of the site and the target audience (Lynch & Horton, 1999; 
Rosenfeld & Morville, 1998). Usability experts stress the important role users play in 
determining if a site fulfills its purpose and meets the needs and expectations of the intended 
audience (Head, 1999; Krug, 2000; Mayhew, 1999; Nielsen, 2000; Spool, 1999). 

Art, Social Studies, History, and Museum Education and the Web 

There is a growing interest in using web-based educational resources in both K-12 and 
higher education. Indeed, the federal government appears to have embraced "the power of the 
Internet for learning" (see Web-based Education Commission, 2000), and state and national 
standards require the integration of technology into the curriculum. Chapter 113, Texas Essential 
Knowledge and Skills for Social Studies (K.15), for example, requires that students be able to 
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apply "critical-thinking skills to organize and use information acquired from a variety of sources 
including electronic technology" (TEA, 2001). The National Standards for Social Studies 
Teachers states that "social studies teaching and learning are powerful when they are integrative" 
and that integration is encouraged when "the teaching makes effective use of technology" 

(NCSS, 2001). Numerous published guides are available to introduce teachers to countless art, 
social studies, and history-related websites (e.g., Braun & Risinger, 1999; Cohen, 1998; Crane, 
2000; Diaz, 1997; Fredericks, 2000; Leu & Leu, 1999; Partin, 1998; Trinkle & Merriman, 2000; 
Willis, 1997). Many guides and articles provide suggestions on how to incorporate these 
resources into the curriculum and use technology to achieve curricular goals (e.g., Boyer, 1999; 
Braun, 1999; Cassutto, 2000; Singleton & Giese, 1999; VanFossen & Shiveley, 2000; Zukas, 
2000). These guides and articles frequently recommend museum websites for use in the 
classroom. 

Although some educators believe that using the Web can enhance the learning 
environment (e.g., Bridges & DeVaull, 1999; Diem, 2000), others argue that the evidence on 
how the Web can promote learning has not been forthcoming (e.g., Garrett, 1997; Owston, 1997; 
Windschitl, 1998). Windschitl emphasizes the need for empirical rather than anecdotal evidence 
concerning the results of Internet-based teaching and learning, and recommends areas for further 
research (e.g., the influence of the Web on independent student inquiry and the role of teachers). 
Berson (1996) and Diem (2000) note that few studies substantiate the positive effects of 
technology in social studies classrooms as an instructional component. Optimistic that the 
evidence will be forthcoming at a future date, Mioduser, Nachmias, Lahav, and Oren (2000) 
nonetheless acknowledge their research indicates that many educational websites currently are 
lacking in pedagogical approaches. Maddux (1996) concludes that, in spite of the immense 
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potential of the Web as a learning tool, the educational value is limited because web authors, 
even educators, seem to ignore the principles of learning. The lack of educational resources as 
well as the quality of information and educational resources available over the Internet also 
concerns educators. Information is not peer referenced or reviewed (Gray, 1999). There are no 
editorial standards nor is anyone in charge of the Internet. The ease with which web authors can 
create and publish pages has resulted in a large number of web resources of questionable quality 
(Barlow, 1998; Cashen, 1995; Cooke, 1999; Doyle & Martorana, 1999; Johnson, 1998). 

Methods 

Participants 

The participants came from several different populations of potential Bayou Bend 
website users, including Bayou Bend docents, K-12 teachers, faculty, graduate students, library 
professionals, and others interested in web-based educational resources. 

Instrument 

This study used a 53-item web-based survey instrument developed specifically to 
measure user perceptions of the design, content, and educational value of the Bayou Bend 
website (see Appendix). The first portion of the survey consisted of a Likert-type rating scale 
(1 = strongly disagree to 4 = strongly agree) for the evaluation criteria previously identified in 
the literature. The survey also included items to measure user perceptions of the educational 
value of components such as the lesson plans and bibliography. The second part of the survey 
asked participants to compare the education section, usability, information, and use of 
multimedia on this site to other art museum websites using a five-point scale (1 = much worse to 
5 = much better). The third part of the survey requested demographic information including 
status, subject expertise, how often participants use the Internet, how comfortable they are using 
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the Internet, how much time they spent reviewing the site before evaluating it, and how likely 
they are to visit the site again. The fourth and final part of the survey provided an opportunity for 
participants to write in comments and suggestions for improvement. A principal components 
factor analysis, conducted to establish the construct validity of Items 1 to 32, revealed five 
factors with eigenvalues greater than 1.0, which represented 57% of the total variance. Table 1 
shows the rotated factor loadings of the 30 items on the survey used to construct the five scales 
named (a) content, (b) educational value, (c) information architecture, (d) graphic design/page 
layout, and (e) search system. Loadings less than the .50 criterion level previously set resulted in 
the elimination of Item 3 (contact information is available) and Item 4 (the purpose of the site is 
apparent) from the scales. Factor loadings for the remaining 30 items ranged from .50 to .79. 
Cronbach's coefficient alpha established internal consistency reliability for each of the factors. 
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Table 1 Factor Loadings of 30 Items on the Bayou Bend Evaluation Survey 











Factors 






Variables 


1 


2 


3 


4 


5 


CONTENT 












1 . 


The organization responsible for the site is an expert on the subject. 


.68 










2. 


The information appears to be up-to-date. 


.65 










5. 


The information adequately covers the subject matter. 


.57 










6. 


The information adheres to high editorial standards. 


.58 










7. 


The information appears to be factual. 


.74 










EDUCATIONAL VALUE 












8. 


Copyright information is available. 




.52 








25. 


The lesson plans are suitable for teaching the subject matter. 




.70 








26. 


The activities for children reinforce the educational experience of an actual 














visit to Bayou Bend. 




.68 








27. 


The Belter Parlor "case study" provides insight into the work of a curator. 




.65 








28. 


The bibliography is helpful for learning more about the subject. 




.72 








29. 


The links to other websites in the "Education" section support the subject matter. 




.79 








30. 


The site supports independent learning. 




.74 








31. 


The depth of the information is adequate for educational purposes. 




.74 








32. 


I would use this site as an educational resource. 




.68 








INFORMATION ARCHITECTURE 












9. 


The information is well organized. 






.60 






10. 


It is easy to find information about the collection. 






.52 






11. 


It is clear where each link will take you. 






.69 






12. 


It is clear which section of the site you are in at all times. 






.70 






13. 


The navigation system for browsing the site is consistent. 






.53 






GRAPHIC DESIGN/PAGE LAYOUT 












17. 


All of the pages have a similar look. 








.55 




18. 


The graphic design supports rather than competes with the display of information. 








.70 




19. 


The page layout makes the site easy to use. 








.60 




20. 


The color scheme is appropriate for the subject matter. 








.66 




21. 


The same basic elements appear in the same location on all pages. 








.58 




22. 


The typefaces used are appropriate for the subject matter. 








.76 




23. 


The information is clearly presented on each page. 








.64 




24. 


There is a good balance between text and images. 








.50 




SEARCH SYSTEM 












14. 


The "Collection Search" makes it clear that only the collection is being searched. 










.50 


15. 


The "Search Help" provides clear examples of how the search system works. 










.69 


16. 


I found what I was looking for using the search system. 










.70 




Eigenvalues 


10.15 


3.71 


1.72 


1.45 


1.22 




% of Variance 


11.20 


15.71 


9.52 


13.35 


7.25 
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Procedures 

Data collection occurred over an eight- week period from February to April 2001 . 
Volunteer participants received invitation letters via the mail, in person, and through email, (i.e., 
online discussion groups for art, social studies, history, educational technology, and 
librarianship) to review the site and submit the online survey. At the end of eight weeks, there 
were 286 responses to the survey. The researcher eliminated 20 responses as follows: 3 blanks, 9 
exact duplicates, 3 questionable responses (e.g., all items rated very low, but rated "much better" 
in the comparison section), and 5 outliers in a one-way ANOVA of each factor. 

Analysis 

Basic descriptive statistics describe the characteristics of the participants as well as the 
survey results. The final data analysis used a three-way ANOVA on each of the five factors. The 
three independent variables were (a) user group, (b) user comfort level, and (c) likelihood of a 
revisit. The five dependent variables were the factors identified through factor analysis: 

(a) content, (b) educational value, (c) information architecture, (d) graphic design/page layout, 
and (e) search system. Pearson Product-Moment Correlations were used to determine the nature 
of the relationships among the items used to evaluate the multimedia components. A review of 
the comments and suggestions for improvement written in by the participants helped to identify 
areas for improvement. 

Results 

The results of this study indicate that the participants had favorable perceptions of the 
design and content but less than favorable perceptions of the educational value. The results also 
indicate that there are statistically significant differences by user group, user comfort level, and 
likelihood of a revisit. There are no statistically significant differences by user experience. 
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Demographic Characteristics 

As shown in Table 2, the final sample of 266 participants consisted of Bayou Bend 
docents (n = 26), library professionals (n = 51), faculty (n = 30), graduate students (n_= 93), 

K-12 teachers (n = 30), and others interested in web-based educational resources (n = 36). 
Because participants were able to select more than one status, for the purpose of this study user 
groups were determined as follows. Bayou Bend docents took precedence over all other groups. 
After the Bayou Bend docents, library professionals (including library directors, library faculty, 
librarians, and archivists) took precedence over the remaining groups. Next, were faculty, 
graduate students, and K-12 teachers, respectively. The "other" group (n = 36) consisted of 
undergraduates (n = 11), museum professionals including museum directors, curators, and 
museum educators (n = 9), technical writers (n = 4), usability experts (n = 4), webmaster (n = 1), 
board assistant (n = 1), horticulturist (n = 1), and online discussion participant (n = 1). Five 
participants did not indicate their status. 



Table 2 


User Group as Determined by Status 




Group 


Frequency 


Percentage 
of Total (%) 


Bayou Bend Docents 


26 


10 


Library Professionals 


51 


19 


Faculty 


30 


11 


Graduate Students 


93 


35 


K-12 Teachers 


30 


11 


Others 


36 


14 




266 


100 
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A majority of the participants reported subject expertise in one of the three subject areas: 
art, social studies, and history. An exact percentage is not available as many of the participants 
reported an expertise in more than one area, but 48% reported subject expertise that was other 
than art, social studies, or history (Table 3). An overwhelming majority (84%) of the participants 
reported using the Internet "daily" (Table 4). Most of the participants said they were "very 
comfortable" (55%) using the Internet (Table 5). Almost half (48%) of the participants said they 
spent more than 30 minutes reviewing the site in order to evaluate it (Table 6). Most of the 
participants also reported that they would be "very likely" (24%) or "likely" (49%) to revisit the 
website (Table 7). 



Table 3 


Area of Subject Expertise 




Subject Area 


Frequency 


Percentage (%) 


Art Education 


36 


14 


Decorative Arts 


36 


14 


Fine Arts 


55 


21 


Social Studies 


34 


13 


U.S. History 


60 


23 


Other 


127 


48 



Note. Totals do not equal 100 as some participants indicated expertise in more than one subject 
area. 



O 

ERLC 
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Table 4 


How Often Respondents Use the Internet 




Internet Usage 


Frequency 


Percentage 
of Total (%) 


Daily 


223 


84 


Weekly 


33 


12 


Monthly 


1 


0 


Almost never 


4 


2 


No response 


5 


2 


Total 


266 


100 



Table 5 


How Comfortable Respondents Are Using the Internet 






Percentage 


Comfort Level 


Frequency 


of Total (%) 


Very comfortable 


145 


55 


Comfortable 


62 


23 


Not comfortable 


2 


1 


Very uncomfortable 


52 


20 


No response 


5 


2 


Total 


266 


101 



Note. Rounded off percentages do not equal 100. 
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Table 6 How Much Time Respondents Spent Reviewing the Website 



Amount of Time 


Frequency 


Percentage 
of Total (%) 


More than an hour 


26 


10 


30 minutes to an hour 


101 


38 


15 to 30 minutes 


107 


40 


Less than 15 minutes 


27 


10 


No response 


5 


2 


Total 


266 


100 



Table 7 


How Likely Respondents Are to Visit the Website Again 


Likely to Revisit Website 


Frequency 


Percentage 
of Total (%) 


Very likely 




65 


24 


Likely 




131 


49 


Not likely 




52 


20 


Very unlikely 




13 


5 


No response 




5 


2 


Total 




266 


100 
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Results for Multimedia Components 

As shown in Table 8, less than half of the participants reported being able to access the 
audio clips (40%) or video clips (44%), while a majority reported being able to access the virtual 
room tour (55%). 



Table 8 



Ability to Access Multimedia Components 



Component 


Frequency 


Percentage 
of Total (%) 


Able to access audio clips 


Yes 


106 


40 


No 


149 


56 


No response 


11 


4 


Total 


266 


100 


Able to access video clips 


Yes 


116 


44 


No 


135 


51 


No response 


15 


6 


Total 


266 


101 


Able to access virtual room tour 


Yes 


147 


55 


No 


106 


40 


No response 


13 


5 


Total 


266 


100 



Note. Rounded off percentages do not equal 100. 
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Tables 9 and 10 show the results of the Pearson Product-Moment Correlations for the audio 
clip and video clip variables, respectively. There is a statistically significant positive very high 
correlation between Add Interest and Enhance the Content for both the audio clip variables and 
the video clip variables (r = .97, p < .001). The standard deviations, which are larger than the 
means, suggest a serious dispersion problem (i.e., there are large differences in the ways 
participants responded to those items). 



Table 9 Results Obtained from Pearson Product-Moment Correlation for Audio 



Add Interest Enhance the Content 



N M SD M SD r p 



266 1.43 1.75 1.38 1.75 .97 <.001 



Note. A score of 4 indicates that participants responded "Strongly Agree" to Item 34 "The audio 
clips add interest to the site" and Item 35 "The audio clips enhance the content of the site." A 
score of 1 indicates that participants responded "Strongly Disagree." 

Table 10 Results Obtained from Pearson Product-Moment Correlation for Video 



Add Interest Enhance the Content 



N M SD M SD r p 



266 1.53 1.76 1.53 1.77 .97 <.001 



Note. A score of 4 indicates that participants responded "Strongly Agree" to Item 37 "The video 
clips add interest to the site" and Item 38 "The video clips enhance the content of the site." A 
score of 1 indicates that participants responded "Strongly Disagree." 
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As shown in Table 11, there is a statistically significant positive very high correlation 
between Adds Interest and Visit Preparation (r = .94, p < .001), between Adds Interest and 
Visitor Attraction (r = .94, p < .001), and between Visit Preparation and Visitor Attraction 
(r = .95, p < .001). The standard deviations, which are almost as high as the means, suggest 
that there are large differences in how participants responded to those items. 



Table 11 Means, Standard Deviations, and Correlations for Virtual Room Tour Variables 





M 


SD 


A 


B 


C 


A. Adds Interest 


2.06 


1.84 


1.00 






B. Visit Preparation 


1.94 


1.82 


94 * * * 


1.00 




C. Visitor Attraction 


1.91 


1.79 


94 * * * 


95 *** 


1.00 



Note. A score of 4 indicates that participants responded "Strongly Agree" to Item 40 "The virtual 
room tour adds interest to the site," Item 41 "The virtual room tour is a good preparation for an 
actual visit," and Item 42 "The virtual room tour will attract visitors to the 'real' Bayou Bend." A 
score of 1 indicates that participants responded "Strongly Disagree" to the items. 

***P< .001. 
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The frequencies shown in Table 12 indicate that many participants did not respond to the 
items regarding the audio clips, video clips, or the virtual room tour. The results suggest that 
most of the participants who were able to access the multimedia components had favorable 
perceptions of the audio and video clips as well as the virtual room tour. 



Table 12 Frequencies of Multimedia Items on the Bayou Bend Evaluation Survey 









Frequencies 




Variables 


0 


1 


2 


3 


4 


AUDIO CLIPS 

34. The audio clips add interest to the site. 


155 


3 


3 


49 


56 


35. The audio clips enhance the content of the site. 


159 


2 


8 


38 


59 


VIDEO CLIPS 

37. The video clips add interest to the site. 


146 


2 


8 


50 


60 


38. The video clips enhance the content of the site. 


147 


3 


8 


45 


63 


VIRTUAL ROOM TOUR 

40. The virtual room tour adds interest to the site. 


113 


2 


6 


45 


100 


4 1 . The virtual room tour is a good preparation for an actual visit. 


119 


3 


9 


46 


89 


42. The virtual room tour will attract visitors to the "real" Bayou Bend. 


118 


4 


11 


51 


82 



Note. 0 — No Response, 1 = Strongly Disagree, 2 = Disagree, 3 = Agree, 4 = Strongly Agree. 



Results for Comparisons to Other Art Museum Websites 

As shown in Table 13, less than half of the participants (48%) thought that the Education 
Section was "better" or "much better" than art other museum websites. The majority of 
participants (56%), however, thought the Usability of the site was better or much better. Most 
participants (54%) also thought the Information was better or much better. Less than half of the 
participants (49%) thought the Use of Multimedia was better or much better. Over 10% of the 
participants did not respond to any of the comparisons to other art museum websites. 
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Table 13 Frequencies for Comparisons to Other Art Museum Websites 





Frequency 


Percentage 
of Total (%) 


Education Section 


Much better 


32 


12 


Better 


96 


36 


About the same 


95 


36 


Worse 


4 


2 


Much worse 


0 


0 


No response 


39 


15 


Total 


266 


101 


Usability 


Much better 


36 


14 


Better 


1 12 


42 


About the same 


82 


31 


Worse 


4 


2 


Much worse 


0 


0 


No response 


32 


12 


Total 


266 


101 


Information 


Much better 


40 


15 


Better 


103 


39 


About the same 


88 


33 


Worse 


1 


0 


Much worse 


1 


0 


No response 


33 


12 


Total 


266 


99 


Use of Multimedia 


Much better 


42 


16 


Better 


87 


33 


About the same 


81 


30 


Worse 


5 


2 


Much worse 


3 


1 


No response 


48 


18 


Total 


266 


100 



Note. Rounded off percentages do not equal 100. 
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Descriptive Results 

As shown in Table 14, on a scale from 1 to 4 (1 = strongly disagree to 4 = strongly 
agree), the mean score for the site overall was 3.23. Content received the highest rating (M = 
3.50), followed by Graphic Design/Page Layout (M = 3.47), Information Architecture (M = 
3.39), and Educational Value (M = 2.91). Search System received the lowest rating (M = 2.89). 
The standard deviations for Educational Value and Search System suggest that there was more 
variation on these scales. Frequencies for each item included in the factors as shown in Table 15 
indicate that more participants did not respond to items for Educational Value and Search 
System. 



Table 14 Means and Standard Deviations for the Five Factors 



Factor 


M 


SD 


Content 


3.50 


0.48 


Educational Value 


2.91 


0.94 


Information Architecture 


3.39 


0.57 


Graphic Design/Page Layout 


3.47 


0.50 


Search System 


2.89 


0.97 


Overall Rating 


3.23 


0.51 




Evaluation of an Educational Website 21 



Table 15 Frequencies of 30 Items on the Bayou Bend Evaluation Survey 



Frequencies 





Variables 


0 


1 


2 


3 


4 


CONTENT 












1 . 


The organization responsible for the site is an expert on the subject. 


0 


2 


7 


97 


160 


2. 


The information appears to be up-to-date. 


0 


3 


18 


92 


153 


5. 


The information adequately covers the subject matter. 


0 


1 


10 


118 


137 


6. 


The information adheres to high editorial standards. 


5 


1 


11 


109 


140 


7. 


The information appears to be factual. 


2 


2 


4 


97 


161 


EDUCATIONAL VALUE 












8. 


Copyright information is available. 


28 


2 


19 


101 


1 16 


25. 


The lesson plans are suitable for teaching the subject matter. 


45 


0 


21 


119 


81 


28. 


The activities for children reinforce the educational experience of an actual 














visit to Bayou Bend. 


41 


3 


15 


111 


96 


29. 


The Belter Parlor "case study" provides insight into the work of a curator. 


61 


4 


15 


112 


74 


28. 


The bibliography is helpful for learning more about the subject. 


36 


1 


13 


105 


111 


29. 


The links to other websites in the "Education" section support the subject matter. 


44 


2 


4 


107 


109 


30. 


The site supports independent learning. 


19 


3 


11 


125 


108 


31. 


The depth of the information is adequate for educational purposes. 


19 


2 


13 


129 


103 


32. 


I would use this site as an educational resource. 


23 


5 


21 


108 


109 


INFORMATION ARCHITECTURE 












9. 


The information is well organized. 


2 


1 


14 


96 


153 


14. 


It is easy to find information about the collection. 


2 


5 


16 


100 


143 


15. 


It is clear where each link will take you. 


2 


2 


24 


117 


121 


16. 


It is clear which section of the site you are in at all times. 


2 


3 


21 


117 


123 


17. 


The navigation system for browsing the site is consistent. 


5 


5 


13 


108 


135 


GRAPHIC DESIGN/PAGE LAYOUT 












17. 


All of the pages have a similar look. 


4 


3 


16 


91 


152 


18. 


The graphic design supports rather than competes with the display of information. 


2 


5 


6 


86 


167 


19. 


The page layout makes the site easy to use. 


0 


2 


12 


102 


150 


20. 


The color scheme is appropriate for the subject matter. 


1 


3 


7 


102 


153 


21. 


The same basic elements appear in the same location on all pages. 


5 


0 


7 


100 


154 


22. 


The typefaces used are appropriate for the subject matter. 


3 


4 


7 


112 


140 


23. 


The information is clearly presented on each page. 


0 


3 


4 


100 


159 


24. 


There is a good balance between text and images. 


2 


7 


29 


102 


126 


SEARCH SYSTEM 












14. 


The "Collection Search" makes it clear that only the collection is being searched. 


16 


2 


17 


126 


105 


15. 


The "Search Help" provides clear examples of how the search system works. 


40 


4 


17 


121 


84 


16. 


I found what I was looking for using the search system. 


30 


12 


33 


105 


86 



Note. 0 = No Response, 1 = Strongly Disagree, 2 = Disagree, 3 = Agree, 4 = Strongly Agree. 



Evaluation of an Educational Website 22 



Three-Way ANOVA Results 

A three-way ANOVA by user group, user comfort level, and likelihood of a revisit 
for each factor revealed statistically significant differences by user group for Educational 
Value (p < .001) and Content and Graphic Design/Page Layout (p < .01), by user comfort 
level for Graphic/Design Page Layout (p < .001), and by likelihood of a revisit for 
Educational Value (p < .001), Content and Graphic Design/Page Layout (p < .01), and 
Information Architecture (p < .05). 

User Group. As shown in Table 16, Library Professionals rated the Content statistically 
significantly higher than did Others. There is no statistically significant difference between 
Library Professionals and Bayou Bend Docents, Faculty, Graduate Students, and K-12 Teachers 
or between Others and Bayou Bend Docents, Faculty, Graduate Students, and K-12 Teachers for 
Content. Library Professionals, Faculty, Graduate Students, and K-12 Teachers rated the 
Educational Value statistically significantly higher than Bayou Bend Docents. There is no 
statistically significant difference between Others and Bayou Bend Docents, Library 
Professionals, Faculty, Graduate Students, and K-12 Teachers for Educational Value. Library 
Professionals rated the Graphic Design/Page Layout statistically significantly higher than did 
Bayou Bend Docents and Others. There is no statistically significant difference between Library 
Professionals, Faculty, Graduate Students, and K-12 Teachers or between Bayou Bend Docents 
and Faculty, Graduate Students, K-12 Teachers, and Others for Graphic Design/Page Layout. 
There is no statistically significant difference between Bayou Bend Docents, Library 
Professionals, Faculty, Graduate Students, K-12 Teachers, and Others for Information 
Architecture or Search System. 
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User Comfort Level. As shown in Table 17, participants who are "very comfortable" 
using the Internet rated the Graphic Design/Page Layout statistically significantly higher than 
those who are "not very comfortable" using the Internet (p < .001). There are no statistically 
significant differences by user comfort level for Content, Educational Value, Information 
Architecture, or Search System. The standard deviations for Educational Value and Search 
System suggest that there was more variation on responses for those factors than the other 
factors. 

Table 17 3-Way ANOVA Summary of Differences of Factors by User Comfort Level 



Very Not Very 

Comfortable Comfortable 

(n = 145) (n= 121) 



Factors 


M 


SD 


M 


SD 


F 


Content 


3.52 


0.44 


3.47 


0.53 


0.037 


Educational Value 


3.03 


0.77 


2.77 


1.09 


2.048 


Information Architecture 


3.44 


0.53 


3.33 


0.62 


0.306 


Graphic Design/Page Layout 


3.59 


0.42 


3.34 


0.56 


10.979*** 


Search System 


3.00 


0.96 


2.76 


0.97 


1.748 



Note. A score of 4 indicates that participants responded "Strongly Agree" to all of the items on the scale. 
A score of 1 indicates that participants responded "Strongly Disagree" to all of the items on the scale. 

***g < .001 . 
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Likelihood of a Revisit. As shown in Table 18, there is a statistically significant 
difference by likelihood of a revisit for Educational Value (2 < .001), for Content and Graphic 
Design/Page Layout (g < .01), and for Information Architecture (g < .05). Participants who are 
"very likely" or "likely" to revisit the site rated Educational Value statistically significantly 
higher than did those "not likely" to revisit. Participants who are "very likely" to revisit the site 
rated Content, Graphic Design/Page Layout, and Information Architecture statistically 
significantly higher than did those "not likely" to revisit. The standard deviations for Educational 
Value and Search System again suggest that there was more variation on responses for those 
factors than the other factors. 



Table 18 3-Way ANOVA Summary of Differences of Factors by Likelihood of a Revisit 



Factors 


Very Likely 
(n = 65) 
M SD 


Likely 
(n= 131) 
M SD 


Not Likely 
(n = 70) 
M SD 


F 


Content 


3.63a 


0.40 


3.50ab 0.47 


3.37b 


0.55 


5.231** 


Educational Value 


3.15a 


0.79 


2.98a 0.91 


2.56b 


1.03 


8.026*** 


Information Architecture 


3.54a 


0.49 


3.37ab 0.53 


3.29b 


0.68 


3.463* 


Graphic Design/Page Layout 


3.62a 


0.44 


3.47ab 0.48 


3.35b 


0.57 


5.752** 


Search System 


2.98 


1.06 


2.94 0.93 


2.72 


0.96 


1.277 



Note. Means with the same letter are not statistically significantly different. A score of 4 indicates that 
participants responded "Strongly Agree" to all of the items on the scale. A score of 1 indicates that 
participants responded "Strongly Disagree" to all of the items on the scale. 

*g < .05. **g< .01. ***g < .001 . 
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Open-Ended Comments and Suggestions for Improvement 

More than half of the participants (n = 158) wrote comments or suggestions for 
improvement to the site. Most comments addressed very specific items covered in the survey. 
Some of the comments were favorable (e.g., "I thought the education portions and student 
activities were excellent"). Many of the comments, however, noted deficiencies or problems 
(e.g., "The lesson plans are not much good unless they are tied to actual school curricula or at 
least have specific learning goals attached"). Several of the participants commented that the 
website made them want to visit the real Bayou Bend Collection and Gardens. 

Information Architecture. Comments regarding elements of the information 
architecture were generally favorable. Many participants were particularly complimentary about 
the organization of the site (e.g., "The site is beautifully organized"). Others suggested 
improving the labels (e.g., "The difference between 'information' and 'about Bayou Bend' isn't 
very clear - 1 had to click on these to see what these included"). 

Search System. Most comments about the search system were not favorable. Participants 
reported that they were unable to find what they were looking for, and some commented that 
they found the search help to be inadequate. Not all comments were unfavorable. One participant 
wrote, "The search tool is outstanding for taking a closer look at particular pieces of the 
collection." 

Content. Most comments regarding the content pertained to the currency and accuracy of 
the information, noting that some of the lecture information was out of date and some of the 
information about the rooms was inaccurate. A number of participants offered suggestions for 
additions to the content such as special events and more information about the rooms, Miss 
Hogg, and the gardens. 
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Graphic Design/Page Layout. Comments regarding elements of the graphic design and 
page layout were generally favorable (e.g., "The layout and design is fantastic"). Suggestions for 
improvement included the use of more color on the site as well as more and larger images. 

Educational Components. Most of the education-related comments pertained to the 
lesson plans. Some comments were favorable (e.g., "The lesson plans are a great resource for 
teachers"). Other comments were not so favorable (e.g., "I, however, would not rate the lesson 
plans particularly high... I'm not sure what students are supposed to know at the end"). Several 
participants recommended more lesson plans and activities on the site as well as information and 
images about the costume presentation the docents do for the fifth grade tours. Comments 
regarding the bibliography and the research guide were generally favorable. 

Multimedia Components. Most comments regarding the audio and video clips and the 
virtual room tour were not favorable. Participants reported technical difficulties with 
downloading the RealPlayer and QuickTime plug-ins. Many participants wrote about problems 
that they had accessing the audio and video clips as well as the virtual tour (e.g., "The virtual 
tour came up as only a strip of the room it meant to show"). Some participants expressed 
frustration while others expressed disappointment at not being able to access these components. 
Participants also noted that download time was a problem. Another concern was the poor quality 
of the video clips and the virtual tour QuickTime movies. Some comments, however, were 
favorable (e.g., "The virtual tour is very well done-great quality on the images, and good 
descriptions"). 




Evaluation of an Educational Website 28 



Discussion 

The results of this evaluation identified several areas for improvement. They also suggest 
a number of implications for practice as well as for further research in the design, development, 
and evaluation of educational websites. 

The first research question this study addressed was "What are users' perceptions of the 
design, content, and educational value of the Bayou Bend website?" The results of the study 
indicate that the participants had generally favorable perceptions of the content, information 
architecture, and graphic design/page layout. This suggests the participants perceived the 
information as good, the structure, navigation, labeling, and graphic design/page layout 
functional. The participants, however, had less than favorable perceptions of the search system, 
multimedia components, and the educational value of the Bayou Bend website. This suggests a 
need for developers to investigate (a) why potential users of the site had poor perceptions of 
these components and (b) how these components might be improved. 

Search System. Although the Search System received the lowest rating of all of the 
factors, a review of the frequencies for the items measuring this scale reveals that a larger 
number of participants did not respond to these items than responded to items measuring the 
other factors, with the exception of Educational Value, which also received a lower rating than 
did the other factors. Most of the participants who responded to the search-related items rated the 
Search System favorably. Written comments regarding the Search System seem to indicate that 
participants expected there to be more items than are currently available in the database (e.g., 
"Why were there only 13 items when I searched 'ceramics', and only 1 item when I searched 'tea 
table'?"). At this writing, there are only 13 ceramic items and only 1 tea table in the database. 
While the Bayou Bend Collection now consists of over 5,000 objects, MFAH administrators 
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have chosen to include only 127 objects in the database on the website. The written comments 
and low rating suggest a need to reevaluate the nature of the items used to measure user 
perceptions of the Search System. In particular, Item 1 6 "I found what I was looking for using 
the search system" should perhaps be changed to see if users can locate a specific object (e.g., "It 
is easy to find the 'Boy with Toy Horse' portrait using the search system"). Items of this nature 
may be a better measure of the functionality of the search system as opposed to content of the 
site. 

Multimedia Components. Participants rated the audio clips and video clips very poorly 
on both variables "add interest" and "enhance the content." A majority of participants, however, 
did not respond or responded that they were unable to access the audio and video clips. A review 
of the frequencies for these items reveals that most of the participants who accessed the audio 
and video clips rated them favorably. This suggests that the choice of audio and video clips is 
appropriate for the Bayou Bend website. The inability of most participants to access the audio 
and video clips indicates a need for developers to explore the reasons why participants were 
unable to access these components (e.g., the clips are difficult to locate on the site or users lack 
Internet expertise). Participants also rated the virtual room tour variables "adds interest," "visit 
preparation," and "visitor attraction" poorly. Again, a large number of participants reported that 
they were unable to access the virtual room tour or did not respond to the item regarding access. 
A review of the frequencies for the virtual tour-related items reveals that most of the participants 
who rated these variables rated them favorably as well. This suggests that the virtual room tour is 
an appropriate multimedia component for the Bayou Bend website. The inability of many 
participants to access the virtual room tour indicates a need for developers to investigate 
problems with accessibility (e.g., long download times or lack of Internet expertise). 
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Educational Value. Participants rated the Educational Value of the Bayou Bend website 
lower than they rated Content, Information Architecture, and Graphic Design/Page Layout, 
however, a review of the frequencies for items measuring this scale reveals that more 
participants did not respond to these items than for any other factor. Most of the participants who 
did respond rated these items favorably although a fair number of participants rated them poorly. 
These results suggest that perhaps many of the participants who responded to the survey were an 
inappropriate audience for evaluating the educational components, such as lesson plans and 
activities for children, either due to a lack of interest or qualifications. The results for those that 
did respond to the education-related items suggest a need to improve the educational components 
of the site. More participants did not respond to the item regarding the usefulness of the Belter 
Parlor "case study" than for any other item measuring a factor. Either a lack of interest in this 
component or an inability to locate the case study on the site may explain the lack of response. 
The case study, which is part of the Research Guide, does not appear until the fourth level of the 
site. Nor does it appear on the site map. 

Comparisons to Other Art Museum Websites. The results of the comparisons indicate 
that the majority of participants thought that the Education Section, Usability, Information, and 
Use of Multimedia on the Bayou Bend website are at least as good as, if not better than, those 
components on other art museum websites. A better and more interesting approach might be to 
have the participants review two or more websites, without letting them know which one is being 
evaluated, and then ask them to compare specific features of the two sites. 

The second research question this study addressed was "Are there statistically significant 
differences in perception by user group, user experience, user comfort level, or likelihood of a 
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revisit?" The results of the study indicate that there are statistically significant differences in user 
perceptions (a) by user group for content, educational value, and graphic design/page layout, 

(b) by user comfort level for graphic design/page layout, and (c) by likelihood of a revisit for 
content, educational value, information architecture, and graphic design/page layout. There are 
no statistically significant differences in user perceptions by user experience. The reasons for 
these differences are not immediately obvious, which suggests a need for further investigation 
regarding user perceptions. 

User Group. Overall, the Bayou Bend Docents and the Others group rated all factors 
lower than the rest of the groups (i.e., Library Professionals, Faculty, Graduate Students, and K- 
12 Teachers). It is conceivable that the docents, many of whom know the collection extremely 
well, might be more critical than would other participants. In addition, the Bayou Bend website 
is representative of an organization with which the docents are very proud to be associated. It is 
reasonable to conclude that they took this evaluation very seriously in order to provide responses 
to help make the site better. It is not evident why both Bayou Bend Docents and Others rated the 
factors differently from the rest of the sample. The results seem to suggest that the Bayou Bend 
Docents and Others are more alike, and the Library Professionals, Faculty, Graduate Students, 
and K-12 Teachers are more alike. Perhaps Library Professionals, Faculty, Graduate Students, 
and K-12 Teachers may have more of an education background that makes them more similar. 

Comfort Level. The participants who reported that they are "very comfortable" rated the 
Graphic Design/Page Layout significantly higher than those who reported that they are "not very 
comfortable” using the Internet. A review of all factor ratings shows that those who are "very 
comfortable" rated all factors higher than did those who are "not very comfortable.” This 
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suggests that perhaps users who feel more comfortable using the Internet also perceive web 
pages more favorably than those who are not very comfortable using the Internet. 

Likelihood of a Revisit. The participants who reported that they would be "very likely" 
to revisit the Bayou Bend website rated all of the factors higher than did those who reported that 
they would be "likely" or "not likely" to revisit. The participants who reported that they would be 
"likely" to revisit also rated all of the factors higher than did those who reported that they would 
be "not likely" to revisit. These results seem to indicate that the perceived quality and usefulness 
of a website may influence whether or not users will return to the site. For example, if users who 
want to use the site for educational purposes perceive the Educational Value as poor when they 
first visit, they are not likely to come back to the site. 

Implications for Practice 

This study suggests a number of implications for practice concerning the design, 
development, and evaluation of web-based educational resources. These include the need for 
(a) user involvement during each phase of development, (b) professional writers and copy 
editors, (c) routine site maintenance, and (d) improvement and further development of the 
educational and multimedia components, graphic design, and search system. In addition, there is 
a need to address the accessibility and usability of the site as well as to provide teacher training 
and technical support to both encourage and aid teachers in the use of the Bayou Bend website as 
an educational resource. 

User Involvement. The results of this study seem to support the value of user 
involvement during each phase of website development as recommended by design and usability 
experts. The survey results demonstrate the useful feedback users can provide developers for 
making improvements to the design and content of the site as well as for guiding future 
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development efforts. Users, for instance, may be in a better position to identify problems and to 
make recommendations for future development than are developers. Bayou Bend docents, who 
rated the content slightly lower than any other group, commented on very specific concerns 
regarding the currency and accuracy of the content. In addition, the docents, who are the subject 
matter experts as well as potential users, were able to identify specific problems with the 
currency of the content of which developers were unaware (e.g., "Staff listing needs updating as 
of 2-1 1-01 with replacement of Nicole by Chad in the BB office"). Bayou Bend docents were 
also able to point out specific inaccuracies in the information (e.g., "There seems to be some 
conflict on the lesson plan site regarding the desk and bookcase. While it shows correctly that 
this is a Newport desk, later it talks about 'this Boston example'"). 

Content. Overall, the participants rated the content favorably in spite of the many errors 
pointed out by the docents in the comment section. Participant comments regarding the 
timeliness and accuracy of the information support Holzschlag's (1998) recommendation to hire 
a professional copy editor to ensure clean copy and a professional writer to ensure the quality of 
the writing. Not only should the information be accurate and interesting, it should be written 
specifically for the Web (Kilian, 1999). This means that information currently on the site taken 
from print sources, including the catalog and brochures, probably should be rewritten for use on 
the website in order to improve readability. Criteria for evaluating the quality of websites include 
the authority and reputation of the author or sponsor (Alexander & Tate, 1999; Cooke, 1999). 
Users, who rely on the reputation of the organization, in this instance the Museum of Fine Arts, 
Houston, to provide reliable information, will most likely assume that information that is 
provided on their website is correct. This suggests that an organization's website should adhere to 
the same high editorial standards as any of their publications. As suggested by the docent who 
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wrote, "I want this presentation to best represent the collection that I love," a website is indeed 
representative of the organization that sponsors it. Participants also commented on the type of 
information they would like to see added to the site (e.g., more lesson plans and activities and 
more information about Miss Hogg and the gardens). 

Educational Value. The Educational Value factor received a low rating, suggesting a 
need for improvement for this aspect of the site. Participants made several good 
recommendations for improvements to the Education Section. One participant suggested, "More 
information on the Costume Presentation in the Education Section." This is an excellent 
recommendation since the same teachers who bring their students for tours of Bayou Bend, 
hopefully, will use the educational resources provided on the site to both prepare for the visit as 
well as to follow up on the visit. The costume presentation is something the students see during 
their tour of Bayou Bend. Some participants suggested adding more lesson plans and activities. 
Others recommended better lesson plans that support specific curricula (e.g., "The lesson plans 
could be a little difficult for new teachers who were not sure of the curriculum requirements of 
their grade"). The implication is that developers should add additional lesson plans and activities 
and involve teachers who will be using the materials in the creation of these materials. Maddux 
(1996) believes that the educational value of most web-based educational resources is limited 
because web authors, even educators, seem to ignore the principles of learning. The results of 
this study suggest a need for developers of web-based educational resources to address 
pedagogical strategies and the principles of learning during development. For instance, 
developers should provide both the learning objectives and grade levels on the lesson plans. 

Some participants did not feel qualified or inclined to respond to some of the education- 
related items. One participant wrote, "The mention of schools reminds me of a suggestion for 
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improvement to the survey. The question about the usefulness of lesson plans is difficult to rate 
for someone who is not a public school teacher." A lower response rate subsequently resulted in 
lower scores overall for Educational Value. While the written comments indicate some concern 
with quality and number of lesson plans, a review of individual scores shows that many of the 
participants who did respond to these items rated the items favorably. This suggests that perhaps 
many members of the sample do not constitute an appropriate audience for evaluating the 
Educational Value of the website. Developers need to identify an appropriate audience qualified 
to evaluate individual special components such as lesson plans and activities for children. 

Graphic Design/Page Layout. Although participants favorably perceived the Graphic 
Design/Page Layout, they wrote in comments and suggestions for improvement that developers 
should carefully consider. Some participants recommended the addition of more and larger 
images and suggested that the smaller images, for instance in the History and Gardens Sections, 
should be clickable for enlarged pictures. Participants also commented that the picture of the 
house on the home page and the map of the gardens are too small for clear viewing. Developers 
should consider enlarging the picture of the house as well as other pictures on the site and 
creating a larger, more legible map of the gardens. In addition, developers should consider 
adding pictures where they will provide interest and add value to the information on the site. 

Search System. The Search System was less than favorably perceived by the participants 
in this study. Although part of this is due to a low response rate for items measuring the Search 
System, the results of the study suggest a need to make improvements to this component of the 
Bayou Bend website. Both the ratings and the written comments indicate that participants were 
unable to locate what they were looking for using the search tool. This may be attributable to the 
fact that most items are not currently in the database. Additionally, it is now more apparent that 
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the search interface is confusing. A browse feature or index may improve search results as well 
as make it clearer to users what objects are available on the site. The survey results also suggest 
developers need to improve the search help. 

Multimedia. The comments of participants suggest a need to improve the quality of the 
video clips (e.g., "The streaming video at the site was blurred and garbled"). Written comments 
also suggest that developers should continue to develop the virtual room tour by adding the rest 
of the rooms and including more hot spots in each room. Many of the participants who were able 
to access the components rated them favorably, indicating that the components add interest and 
enhance the content. This suggests that the choice of multimedia components is appropriate for 
the purpose and audience for this website. Experts recommend the use of multimedia only if it 
adds interest and/or enhances the content (Cooke, 1999; Lynch & Horton, 1999). Developers, 
however, need to be concerned about the inability of many of the participants to access the 
multimedia components and should explore this apparent weakness in the site. 

Accessibility. The results of the study pertaining to the multimedia components suggest 
that designers and developers need to be more cognizant of accessibility issues and to develop 
more accessible websites. This is especially important during the development of web-based 
educational resources, including supplemental online course materials or for distance learning. 
Lynch and Horton (1999) remind developers to consider how the site will function for those who 
do not have the best equipment, the most current software, and speedy Internet connections. As 
recommended by Willmarth (1998), designers should identify the target audience hardware just 
as they identify the target audience and design accordingly for that level of machine. Participants 
in this study commented on slow download times suggesting that many of them do not have fast 
Internet connections or state-of-the-art equipment and software. 
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The literature also emphasizes the need to consider physically disabled users, particularly 
those who are blind or visually impaired and may rely on assistive technology devices such as 
speech output devices that read the text from the computer screen. This study did not specifically 
address accessibility, but the inability of many participants to access the audio and video clips 
and the virtual room tour emphasizes the need to address accessibility issues during all phases of 
web design. Web-based educational resources should be accessible to all users. 

Usability. The results of this study support the recommendation of experts to conduct 
usability testing during development (Mayhew, 1999; Nielsen, 2000). Some participants 
commented on problems and frustrations with the navigation and search systems. Rating scales 
and written comments cannot adequately address these concerns. Usability testing in which 
developers ask users to perform tasks (e.g., download a plug-in or search for an object in the 
collection) can help developers identify and resolve potential problems. Developers can observe 
users, asking them to verbalize their thought processes as they navigate the site to find 
information, search for art objects or plants using the collection and garden search systems, or 
access the multimedia components. While the written comments on the survey are helpful, it is 
difficult to tell exactly what kind of problems users are having when they write comments such 
as "I couldn't download real player or audio. Don't know why it wouldn't download." Being able 
to observe and talk to users about these problems will help developers correct them. 

Teacher Training and Technical Support. Quality web-based educational resources are 
not enough. If teachers are successfully going to integrate technology into the curriculum and 
their teaching activities, they need to receive the proper training and the necessary technical 
support to be able to so do. Many of the comments discussed above, although not identified 
specifically with K-12 teachers, suggest a general lack of technical expertise among potential 
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Bayou Bend website users. Teacher training and technical support, provided through hands-on 
workshops, can give teachers practice in using the website and provide ideas on how the site may 
be used, for instance, as a preparation for a first-time visit or follow-up visit. 

Routine Site Maintenance. The development of a website is not a one-time project. 
Someone must take responsibility for maintaining the website in order for the site to remain a 
viable, credible resource. For art museum websites, in particular, which provide timely 
information about upcoming and current exhibits, special events, lectures, or a calendar, it is 
critical to update that information on a routine basis. A good website also requires a certain 
amount of ongoing development, such as the addition of new information, new features, or a 
fresh new graphic design, to encourage repeat visits to the site. New information and an 
occasional new look will assure users that someone is maintaining the site. This, in turn, adds to 
the interest and credibility of the site. 

Implications for Further Research 

The results of this study suggest a need for further research concerning a number of 
different issues pertaining to website evaluation. Studies on (a) website visitor characteristics, (b) 
user design preferences, (d) accessibility and usability, (e) effectiveness of educational materials, 
and (f) collaborative website development, for instance, may provide further insight into the 
design, development, and evaluation of quality web-based educational resources. Survey 
research is helpful as an initial investigation of many of these issues but, as demonstrated by the 
comments from the Bayou Bend evaluation, qualitative methods may prove very useful for 
identifying and dealing with specific concerns. 

Visitor Studies. If developers are designing for the users, as they should, there is a need 
to explore the characteristics of website users. Who are these users? Why did they visit the 
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website? What parts of the website did they visit? Did they have a satisfactory experience? What 
did they like? What did they not like? In addition, what kind of computers are visitors using? 
What type of Internet connection do they have? An online survey placed directly on the website 
can help to gather information about visitors to the site. Studies using log analysis may help to 
answer questions of where users come from, what pages they visit, and how long they stay. 

User Design Preferences. Little research exists for user preferences in graphic design, 
page layout, navigation, and search interfaces on web pages, yet these preferences may affect 
user perceptions. The Graphic Design/Page Layout on the Bayou Bend website was favorably 
perceived by most of the participants, however, some participants wrote in comments expressing 
their preferences for typeface, font size, image size, and background color. Comments about 
typefaces and font sizes suggest that developers should leave typefaces and font sizes in the 
control of users. Browsers allow users to choose their preferred typeface and font size in the 
preferences menu unless developers override user preferences in the HTML code or with 
cascading style sheets. This is also an accessibility issue if users cannot increase small font sizes 
for easy viewing. Surveys and experiments on user design preferences may help developers to 
make appropriate design choices for web-based educational resources. 

Accessibility and Usability. A content analysis of web-based educational resources 
using the Bobby accessibility software has the potential to reveal the most prevalent accessibility 
problems and to identify potential remedies for the problems (CAST, 2000). This information 
would be useful to web developers as they attempt to design and develop accessible websites. 
Formal usability studies can help to determine accessibility and to identify potential problems 
and concerns with the navigation, collection and garden search systems, multimedia components, 
and site map. Survey research can initially investigate accessibility and usability issues (e.g., Are 
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users able to access and use the information on the website?). Follow-up questionnaires and 
interviews with users may provide information that will allow developers to address specific 
weaknesses and concerns. Qualitative methods will allow developers to observe how users 
navigate or search, for instance, and to describe verbally the processes and problems users 
encounter. Studies are needed to discover why users have problems accessing multimedia 
components (e.g., expertise, equipment, or Internet connection speed). As educational 
institutions comply with recent legislation regarding accessibility (see Federal Information 
Technology Accessibility Initiative, 2001; Rehabilitation Act Amendments of 1998), studies to 
examine procedures and policies for compliance may prove beneficial to administrators and 
developers of educational websites. 

Effectiveness of Educational Materials. Experimental studies using pre-tests and post- 
tests may gauge the effectiveness of lesson plans and other educational materials and activities 
based on art objects from the Bayou Bend Collection to teach the curriculum. Experimental 
studies using control groups may test the effectiveness of the virtual room tour on the website 
versus an actual Bayou Bend tour. This information could be useful for art, history, and social 
studies teachers who are unable to bring their students to Bayou Bend. 

Collaborative Website Development. Finally, ethnographic or case studies on the 
collaborative development of educational websites may provide insight into the processes and 
challenges encountered during such projects that will help developers to have more successful 
collaborative experiences and to develop better websites. 

Limitations of the Study 

The final sample consisted of an extremely diverse population of potential Bayou Bend 
website users, which included very small groups of Bayou Bend docents, K-12 teachers, faculty, 
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graduate students, librarians and a group of other participants who responded to the survey 
because of their interest in web-based educational resources. The participants were not actual 
users of the Bayou Bend website, but rather volunteers solicited for the study because they are 
members of the targeted educational audience for the site, which makes them different from 
others who may use the Bayou Bend website. 

Over 1,000 Bayou Bend docents, K-12 teachers, faculty, and students received invitations 
to participate in the evaluation by mail or in person during a presentation about the study. 
Thousands more were invited to participate through a number of online discussion groups on the 
topics of art, social studies, history, educational technology, and librarianship, but only a very 
small number responded to the request for participation. The low response rate and the mixed 
sample limit the generalizability of this study. In spite of these limitations, evaluative studies 
such as this can provide useful information for the development of quality web-based 
educational resources. 
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Appendix 
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Please rate these items in accordance with the following scale: 

1 = Strongly disagree 2 = Disagree 3 “Agree 4 = Strongly agree 
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1. The organization responsible for the site is an expert on the subject. 


o 


o 


o 


o 


2. The information appears to be up-to-date. 


o 


o 


o 


o 


3. Contact information is available. 


o 


o 


o 


o 


4. The purpose of the site is apparent. 


o 


o 


o 


o 


5. The information adequately covers the subject matter. 


o 


o 


o 


o 


6. The information adheres to high editorial standards. 


o 


o 


o 


o 


7. The information appears to be factual. 


o 


o 


o 


o 


8. Copyright information is available. 


o 


o 


o 


o 


9. The information is well organized. 


o 


o 


o 


o 


10. It is easy to find infonnation about the collection. 


o 


o 


o 


o 


1 1 . It is clear where each link will take you. 


o 


o 


o 


o 


12. It is clear which section of the site you are in at all times. 


o 


o 


o 


o 


13. The navigation system for browsing the site is consistent. 


o 


o 


o 


o 


14. The "Collection Search" makes it clear that only the collection is being searched. 


o 


o 


o 


o 


13. The "Search Help" provides clear examples of how the search system works. 


o 


o 


o 


o 


16. 1 found what I was looking for using the search system. 


o 


o 


o 


o 


17. All of the pages have a similar look. 


o 


o 


o 


o 


18. The graphic design supports rather than competes with the display of information. 


o 


o 


o 


o 


19. The page layout makes the site easy to use. 


o 


o 


o 


o 


20, The color scheme is appropriate for the subject matter. 


o 


o 


o 


o 


21 . The same basic elements appear in the same location on all pages. 


o 


o 


o 


o 


22. The typefaces used are appropriate for the subject matter. 


o 


o 


o 


o 


23. The infonnation is clearly presented on each page. 


o 


o 


o 


o 


24. There is a good balance between text and images. 


o 


o 


o 


o 


25. The lesson plans are suitable for teaching the subject matter. 


o 


o 


o 


o 


26. The activities for children reinforce the educational experience of an actual visit to Bayou Bend. 


o 


o 


o 


o 


27. The Belter Parlor "case study" provides insight into the work of a curator. 


o 


o 


o 


o 


28. The bibliography is helpful for learning more about the subject. 


o 


o 


o 


o 


29. The links to other Web sites in the "Education" section support the subject matter. 


o 


o 


o 


o 


30. The site supports independent learning. 


o 


o 


o 


o 


31. The depth of the infonnation is adequate for educational purposes. 


o 


o 


o 


o 


32. 1 would use this site as an educational resource. 


o 


0 


0 


0 
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33. Were you able to access the audio clips? 

O Yes 

O No 

If yes, please rate the following. If no, please proceed to item 36. 

34. The audio clips add interest to the site. O O O O 

35. The audio clips enhance the content of the site. O O O O 

36. Were you able to access the video clips? 

O Yes 

O No 

If yes, please rate the following. If no , please proceed to item 39. 



37. The video clips add interest to the site. 


o 


o 


o 


o 


38. The video clips enhance the content of the site. 


o 


o 


o 


o 



39. Were you able to access the virtual room tour? 

O Yes 
O No 

If yes, please rate the following. If no, please proceed to item 43. 



40. The virtual room tour adds interest to the site. 




o 


o 


o 


o 


41 . The virtual room tour is a good preparation for an actual visit. 




o 


o 


o 


o 


42. The virtual room tour will attract visitors to the "real" Bayou Bend Collection and Gardens. 




o 


o 


o 


o 


Please rate these items in accordance with the following scale: 












1 « Much worse 2- Worse 3 « About the same 4 = Better 


5 “ Much better 
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43. Compared to other art museum Web sites, the education section of this site is 


o 


o 


o 


o 


o 


44. Compared to other art museum Web sites, the usability of this site is 


o 


o 


o 


o 


o 


45. Compared to other art museum Web sites, the information available on this site is 


o 


o 


o 


o 


o 


46. Compared to other art museum Web sites, the use of multimedia on this site is 


o 


o 


o 


o 


o 



Please tell us about you. 

47. Which of the following best describes your status? Please check all that apply. 
I I Bayou Bend Docent 
CD K-12 Teacher 
CD College or University Faculty 
CD Undergraduate 
CD Graduate Student 

CD Other (Please describe in the box below.) 
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48. Which of the following best describes your area of subject expertise? Please check all that apply. 

□ Art Education 
IH Decorative Arts 

□ Fine Arts 

□ Social Studies Education 
n U.S. History 

D Other (Please describe in the box below.) 



49. Which of the following best describes how often you use the Internet? 

O Almost never 

O Monthly 
O Weekly 
O Daily 

50. How comfortable are you with using the Internet? 

O Veiy uncomfortable 

O Hot comfortable 
O Comfortable 
O Very comfortable 

51 . Approximately how much time did you spend looking at the Bayou Bend Web site in order to evaluate it? 
O Less than 15 minutes 

O 15 to 30 minutes 
O 30 minutes to an hour 
O More than an hour 

52. How likely are you to visit the Bayou Bend Web site again? 

O Very unlikely 

O Hot likely 
O Likely 
O Very likely 

53. Comments or suggestions for improvement to the site: 



Submit 



Clear 
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